Ensemble Square Root Kalman Filter Research Articles

Implementation and Testing of Radar Data Assimilation Capabilities Within the Joint Effort for Data Assimilation Integration Framework With Ensemble Transformation Kalman Filter Coupled With FV3‐LAM Model

AbstractCapabilities to directly assimilate radar data are implemented within the local ensemble transform Kalman filter (LETKF) and the gain‐form LETKF (LGETKF) algorithms of the Joint Effort for Data assimilation Integration (JEDI) system. The capabilities are evaluated for the analysis and forecast of a severe convection case of 20 May 2019 in the Southern Great Plains using the limited area model version of the FV3 dynamical core (FV3‐LAM) from a recent release for Short‐Range Weather Application (SRW App). The LETKF and LGETKF implementations are shown to produce analyses and short‐range forecasts comparable to those using the ensemble square‐root Kalman Filter (EnSRF) within the Gridpoint Statistical Interpolation (GSI) framework used by current NCEP operational models. In addition, LGETKF retaining only 60% variances for model‐space vertical localization performs similarly to LGETKF retaining 99% of variance and LETKF using observation error‐based vertical localization. JEDI LETKF shows better parallel scalability than LGETKF and GSI EnSRF.

Bias-aware thermoacoustic data assimilation

Ensemble data assimilation algorithms combine experimental data and numerical models to estimate the state and parameters of a system. If the model is unbiased, the estimation concentrates around the true state. Thermoacoustic instabilities are, however, commonly modelled with low-order models, which are biased by definition. We propose the introduction of reservoir computing to represent the model bias. We combine the ensemble square-root Kalman filter with an echo state network to perform, in real time, (1) the estimation of the state of the system, (2) parameter calibration, and (3) model bias estimation. The proposed methodology is tested in a Rijke tube system, with synthetic experimental data from a high order model.

Estimation of Thrombocyte Concentrate (TC) in PMI Gresik using unscented and square root Ensemble Kalman Filter

Ministry of Health established the Indonesian Red Cross (PMI) as an organization to anticipate the need for blood by obligation to provide enough proper blood for use. Its process of delivering information related to blood donation requires the public always come to PMI. In such way it is less efficient and effective. In addition, there is no system for estimating blood stock to anticipate the blood supply needed in the next period. Blood transfusion is needed in terms of both quality and quantity by patients suffering from various health problems. Due to the urgency of blood transfusion, maintaining the stability of blood stock is a must so as not to cause blood loss due to excessive of the blood stock. To minimize such loss, blood stock prediction is needed. The objective of this study was to estimated the blood demand for blood type of Thrombocyte Concentrate (TC) or concentrated red blood cells at PMI Gresik by applying the method of Unscented Kalman Filter (UKF) and Square Root Ensemble Kalman Filter (SR-EnKF). The simulation results showed that both methods have high accuracy with an error of less than 3%.. The best simulation showed that the error between the real data and the simulation with SR-EnKF was in the order of 0.0023574 with generated 300 ensembles. Where as that with UKF was some 0.025566.

A simultaneous calibration and data assimilation (C/DA) to improve NRLMSISE00 using thermospheric neutral density (TND) from space-borne accelerometer measurements

SUMMARY Improving thermospheric neutral density (TND) estimates is important for computing drag forces acting on low-Earth-orbit (LEO) satellites and debris. Empirical thermospheric models are often used to compute TNDs for the precise orbit determination experiments. However, it is known that simulating TNDs are of limited accuracy due to simplification of model structure, coarse sampling of model inputs and dependencies to the calibration period. Here, we apply TND estimates from accelerometer measurements of the Challenging Minisatellite Payload (CHAMP) and the Gravity Recovery and Climate Experiment (GRACE) missions as observations to improve the NRLMSISE-00 model, which belongs to the mass spectrometer and incoherent scatter family of models. For this, a novel simultaneous calibration and data assimilation (C/DA) technique is implemented that uses the ensemble Kalman filter and the ensemble square-root Kalman filter as merger. The application of C/DA is unique because it modifies both model-derived TNDs, as well as the selected model parameters. The calibrated parameters derived from C/DA are then used to predict TNDs in locations that are not covered by CHAMP and GRACE orbits, and forecasting TNDs of the next day. The C/DA is implemented using daily CHAMP- and/or GRACE-TNDs, for which compared to the original model, we find 27 per cent and 62 per cent reduction of misfit between model and observations in terms of root mean square error and Nash coefficient, respectively. These validations are performed using the observations along the orbital track of the other satellite that is not used in the C/DA during 2003 with various solar activity. Comparisons with another empirical model, that is, Jacchia-Bowman, indicate that the C/DA results improve these quality measurements on an average range of 50 per cent and 60 per cent, respectively.

Navigation and guidance control system of UNUSAITS AUV based on dynamical system using ensemble kalman filter square root

This study presents the development of navigation system of AUV. It is initially in the form of a nonlinear system model to determine the trajectory for the AUV motion control. The non-linear system model of UNUSAITS AUV is then implemented in the Square Root Ensemble Kalman filter (SR-EnKF) to estimate the trajectory of the AUV. The developed EnKF algorithm covers two types of simulations. The contribution from this paper is trajectory or position estimation of nonlinear UNUSAITS AUV system with 6-DOF AUV model. The result of model estimation is presented based on numeric study and simulation. The first simulation is dedicated to generate 400 ensembles, and the second simulation is to generate 400 ensembles. The second simulation shows that the 500 ensembles give more accurate results. A further examination on the accuracy has been performed by accounting for the RMSE of actual condition and the estimation from the simulation, which yields the range of accuracy between 97 % up to 99%.

Considering the impact of observation error correlation in ensemble square-root Kalman filter

Data assimilation has been developed into an effective technology that can utilize a large number of multisource unconventional data. It cannot only provide the initial field for the ocean numerical prediction model, but also construct the ocean reanalysis datasets and provide the design basis for the ocean observation plan. In data assimilation, the estimation of the observation error is of paramount importance, because the quality of the analysis depends on it. In general, the observation error covariance matrix is diagonal or assumed to be diagonal, which means that the observation errors are independent from one another. However, there are indeed correlations in the observation errors. A diagnostic method has been developed, which can estimate a correlated and more accurate observation error covariance matrix. The proposed method combines an ensemble squareroot Kalman filter with the diagnostic method, providing an estimation of the observation error covariance matrix. In order to test the performance of the method, the numerical experiments are performed with the Lorenz 96 model and a Shallow water model. The more accurate observation error covariance matrix can be obtained to use in ensemble square-root Kalman filter by using the new method. We could find using the estimated correlated observation error in the data assimilation improves the analysis.

Parallel Direct Solution of the Ensemble Square Root Kalman Filter Equations with Observation Principal Components

AbstractThe ensemble square root Kalman filter (ESRF) is a variant of the ensemble Kalman filter used with deterministic observations that includes a matrix square root to account for the uncertainty of the unperturbed ensemble observations. Because of the difficulties in solving this equation, a serial approach is often used where observations are assimilated sequentially one after another. As previously demonstrated, in implementations to date the serial approach for the ESRF is suboptimal when used in conjunction with covariance localization, as the Schur product used in the localization does not commute with assimilation. In this work, a new algorithm is presented for the direct solution of the ESRF equations based on finding the eigenvalues and eigenvectors of a sparse, square, and symmetric positive semidefinite matrix with dimensions of the number of observations to be assimilated. This is amenable to direct computation using dedicated, massively parallel, and mature libraries. These libraries make it relatively simple to assemble and compute the observation principal components and to solve the ESRF without using the serial approach. They also provide the eigenspectrum of the forward observation covariance matrix. The parallel direct approach described in this paper neglects the near-zero eigenvalues, which regularizes the ESRF problem. Numerical results show this approach is a highly scalable parallel method.

Improving Large-Domain Convection-Allowing Forecasts with High-Resolution Analyses and Ensemble Data Assimilation

Analyses with 20-km horizontal grid spacing were produced from continuously cycling three-dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and “hybrid” variational–ensemble data assimilation (DA) systems over a domain spanning the conterminous United States. These analyses initialized 36-h Weather Research and Forecasting Model forecasts containing a large convection-allowing 4-km nested domain, where downscaled 20-km 3DVAR, EnSRF, and hybrid analyses initialized the 4-km forecasts. Overall, hybrid analyses initialized the best 4-km precipitation forecasts. Furthermore, whether 4-km precipitation forecasts could be improved by initializing them with true 4-km analyses was assessed. As it was computationally infeasible to produce 4-km continuously cycling ensembles over the large 4-km domain, several “dual-resolution” hybrid DA configurations were adopted where 4-km backgrounds were combined with 20-km ensembles to produce 4-km hybrid analyses. Additionally, 4-km 3DVAR analyses were produced. In both hybrid and 3DVAR frameworks, initializing 4-km forecasts with true 4-km analyses, rather than downscaled 20-km analyses, yielded superior precipitation forecasts over the first 12 h. Differences between forecasts initialized from 4-km and downscaled 20-km hybrid analyses were smaller for 18–36-h forecasts, but there were occasionally meaningful differences. Continuously cycling the 4-km backgrounds and using static background error covariances with larger horizontal length scales in the hybrid led to better forecasts. All hybrid-initialized forecasts, including those initialized from downscaled 20-km analyses, were more skillful than forecasts initialized from 4-km 3DVAR analyses, suggesting the analysis method was more important than analysis resolution.

Assimilating aerosol observations with a “hybrid” variational‐ensemble data assimilation system

AbstractTotal 550 nm aerosol optical depth, surface fine particulate matter (PM2.5), and meteorological observations were assimilated with continuously cycling three‐dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and hybrid variational‐ensemble data assimilation systems. The hybrid system's background error covariances (BECs) were a blend of those in 3DVAR and produced by the cycling EnSRF system, and the 3DVAR, EnSRF, and hybrid systems differed almost exclusively by their BECs. New analyses were produced every 6 h between 0000 UTC 1 June and 1800 UTC 14 July 2010 over a domain encompassing the contiguous United States (CONUS) and adjacent areas. Additionally, a control experiment that only assimilated meteorological observations was performed. Each 1800 UTC analysis initialized a 48 h Weather Research and Forecasting with Chemistry model forecast. These forecasts were evaluated with a focus on air quality prediction. The ensemble aerosol spread was generally insufficient, particularly over the western CONUS. However, despite the suboptimal ensemble spread, the hybrid system performed quite well and usually produced the best aerosol forecasts. Additionally, both the 3DVAR‐ and EnSRF‐initialized forecasts typically outperformed the control. These results are encouraging and suggest the resiliency of the hybrid method. Improved aerosol ensembles should translate into even better future hybrid forecasts.

Convection-Permitting Forecasts Initialized with Continuously Cycling Limited-Area 3DVAR, Ensemble Kalman Filter, and “Hybrid” Variational–Ensemble Data Assimilation Systems

Abstract Analyses with 20-km horizontal grid spacing were produced from parallel continuously cycling three-dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and “hybrid” variational–ensemble data assimilation (DA) systems between 0000 UTC 6 May and 0000 UTC 21 June 2011 over a domain spanning the contiguous United States. Beginning 9 May, the 0000 UTC analyses initialized 36-h Weather Research and Forecasting Model (WRF) forecasts containing a large convection-permitting 4-km nest. These 4-km 3DVAR-, EnSRF-, and hybrid-initialized forecasts were compared to benchmark WRF forecasts initialized by interpolating 0000 UTC Global Forecast System (GFS) analyses onto the computational domain. While important differences regarding mean state characteristics of the 20-km DA systems were noted, verification efforts focused on the 4-km precipitation forecasts. The 3DVAR-, hybrid-, and EnSRF-initialized 4-km precipitation forecasts performed similarly regarding general precipitation characteristics, such as timing of the diurnal cycle, and all three forecast sets had high precipitation biases at heavier rainfall rates. However, meaningful differences emerged regarding precipitation placement as quantified by the fractions skill score. For most forecast hours, the hybrid-initialized 4-km precipitation forecasts were better than the EnSRF-, 3DVAR-, and GFS-initialized forecasts, and the improvement was often statistically significant at the 95th percentile. These results demonstrate the potential of limited-area continuously cycling hybrid DA configurations and suggest additional hybrid development is warranted.

A Regional GSI-Based Ensemble Kalman Filter Data Assimilation System for the Rapid Refresh Configuration: Testing at Reduced Resolution

Abstract A regional ensemble Kalman filter (EnKF) system is established for potential Rapid Refresh (RAP) operational application. The system borrows data processing and observation operators from the gridpoint statistical interpolation (GSI), and precalculates observation priors using the GSI. The ensemble square root Kalman filter (EnSRF) algorithm is used, which updates both the state vector and observation priors. All conventional observations that are used in the operational RAP GSI are assimilated. To minimize computational costs, the EnKF is run at ⅓ of the operational RAP resolution or about 40-km grid spacing, and its performance is compared to the GSI using the same datasets and resolution. Short-range (up to 18 h, the RAP forecast length) forecasts are verified against soundings, surface observations, and precipitation data. Experiments are run with 3-hourly assimilation cycles over a 9-day convectively active retrospective period from spring 2010. The EnKF performance was improved by extensive tuning, including the use of height-dependent covariance localization scales and adaptive covariance inflation. When multiple physics parameterization schemes are employed by the EnKF, forecast errors are further reduced, especially for relative humidity and temperature at the upper levels and for surface variables. The best EnKF configuration produces lower forecast errors than the parallel GSI run. Gilbert skill scores of precipitation forecasts on the 13-km RAP grid initialized from the 3-hourly EnKF analyses are consistently better than those from GSI analyses.

Assimilation of IASI partial tropospheric columns with an Ensemble Kalman Filter over Europe

Abstract. Partial lower tropospheric ozone columns provided by the IASI (Infrared Atmospheric Sounding Interferometer) instrument have been assimilated into a chemistry-transport model at continental scale (CHIMERE) using an Ensemble Square Root Kalman Filter (EnSRF). Analyses are made for the month of July 2007 over the European domain. Launched in 2006, aboard the MetOp-A satellite, IASI shows high sensitivity for ozone in the free troposphere and low sensitivity at the ground; therefore it is important to evaluate if assimilation of these observations can improve free tropospheric ozone, and possibly surface ozone. The analyses are validated against independent ozone observations from sondes, MOZAIC1 aircraft and ground based stations (AIRBASE – the European Air quality dataBase) and compared with respect to the free run of CHIMERE. These comparisons show a decrease in error of 6 parts-per-billion (ppb) in the free troposphere over the Frankfurt area, and also a reduction of the root mean square error (respectively bias) at the surface of 19% (33%) for more than 90% of existing ground stations. This provides evidence of the potential of data assimilation of tropospheric IASI columns to better describe the tropospheric ozone distribution, including surface ozone, despite the lower sensitivity. The changes in concentration resulting from the observational constraints were quantified and several geophysical explanations for the findings of this study were drawn. The corrections were most pronounced over Italy and the Mediterranean region, we noted an average reduction of 8–9 ppb in the free troposphere with respect to the free run, and still a reduction of 5.5 ppb at ground, likely due to a longer residence time of air masses in this part associated to the general circulation pattern (i.e. dominant western circulation) and to persistent anticyclonic conditions over the Mediterranean basin. This is an important geophysical result, since the ozone burden is large over this area, with impact on the radiative balance and air quality. 1 Measurements of OZone, water vapour, carbon monoxide and nitrogen oxides by in-service AIrbus airCraft (http://mozaic.aero.obs-mip.fr/web/).

Application and comparison of Kalman filters for coastal ocean problems: An experiment with FVCOM

Twin experiments were made to compare the reduced rank Kalman filter (RRKF), ensemble Kalman filter (EnKF), and ensemble square‐root Kalman filter (EnSKF) for coastal ocean problems in three idealized regimes: a flat bottom circular shelf driven by tidal forcing at the open boundary; an linear slope continental shelf with river discharge; and a rectangular estuary with tidal flushing intertidal zones and freshwater discharge. The hydrodynamics model used in this study is the unstructured grid Finite‐Volume Coastal Ocean Model (FVCOM). Comparison results show that the success of the data assimilation method depends on sampling location, assimilation methods (univariate or multivariate covariance approaches), and the nature of the dynamical system. In general, for these applications, EnKF and EnSKF work better than RRKF, especially for time‐dependent cases with large perturbations. In EnKF and EnSKF, multivariate covariance approaches should be used in assimilation to avoid the appearance of unrealistic numerical oscillations. Because the coastal ocean features multiscale dynamics in time and space, a case‐by‐case approach should be used to determine the most effective and most reliable data assimilation method for different dynamical systems.

Review of the Nonlinear Filters in the Land Data Assimilation

The land data assimilation research has become the emerging domain in the geoscience,the data assimilation algorithm obtained rapid development and widespread application taking the nonlinear filter as representative's.The extended Kalman filter,unscented Kalman filter,the ensemble Kalman filter and the SIR particle filter are discussed in the Bayesian theory framework from the viewpoint of the recursive Bayesian estimation.Towards the problems in the application of the ensemble Kalman filter and the SIR particle filter,some techniques that can improve the filter performances are also reviewed,such as the covariance localization,the covariance inflation,the double ensemble Kalman filter,the perturbations in the ensembles,the model forcing and parameters,the ensemble square root Kalman filter and the improved variants of the particle filters.The advantages and disadvantages of each filter as well as the applied perspective and the future research directions are discussed.

Assimilation of Simulated Polarimetric Radar Data for a Convective Storm Using the Ensemble Kalman Filter. Part II: Impact of Polarimetric Data on Storm Analysis

Abstract A data assimilation system based on the ensemble square-root Kalman filter (EnSRF) is extended to include the additional capability of assimilating polarimetric radar variables. It is used to assess the impact of assimilating additional polarimetric observations on convective storm analysis in the Observing System Simulation Experiment (OSSE) framework. The polarimetric variables considered include differential reflectivity ZDR, reflectivity difference Zdp, and specific differential phase KDP. To simulate the observational data more realistically, a new error model is introduced for characterizing the errors of the nonpolarimetric and polarimetric radar variables. The error model includes both correlated and uncorrelated error components for reflectivities at horizontal and vertical polarizations (ZH and ZV, respectively). It is shown that the storm analysis is improved when polarimetric variables are assimilated in addition to ZH or in addition to both ZH and radial velocity Vr. Positive impact is largest when ZDR, Zdp, and KDP are assimilated all together. Improvement is generally larger in vertical velocity, water vapor, and rainwater mixing ratios. The rainwater field benefits the most while the impacts on horizontal wind components and snow mixing ratio are smaller. Improvement is found at all model levels even though the polarimetric data, after the application of thresholds, are mostly limited to the lower levels. Among ZDR, Zdp, and KDP, ZDR is found to produce the largest positive impact on the analysis. It is suggested that ZDR provides more independent information than the other variables. The impact of polarimetric data is also expected to be larger when they are used to retrieve drop size distribution parameters. The polarimetric radar data thresholding prior to assimilation is found to be necessary to minimize the impact of noise. This study is believed to be the first to directly assimilate (simulated) polarimetric data into a numerical model.

Simultaneous Estimation of Microphysical Parameters and Atmospheric State with Simulated Radar Data and Ensemble Square Root Kalman Filter. Part II: Parameter Estimation Experiments

Abstract The ensemble Kalman filter method is applied to correct errors in five fundamental microphysical parameters that are closely involved in the definition of drop/particle size distributions of microphysical species in a commonly used single-moment ice microphysics scheme, for a model-simulated supercell storm, using radar data. The five parameters include the intercept parameters for rain, snow, and hail/graupel and the bulk densities of hail/graupel and snow. The ensemble square root Kalman filter (EnSRF) is employed for simultaneous state and parameter estimation. The five microphysical parameters are estimated individually or in different combinations starting from different initial guesses. A data selection procedure based on correlation information is introduced, which combined with variance inflation, effectively avoids the collapse of the spread of parameter ensemble, hence filter divergence. Parameter estimation results demonstrate, for the first time, that the ensemble-based method can be used to correct model errors in microphysical parameters through simultaneous state and parameter estimation, using radar reflectivity observations. When error exists in only one of the microphysical parameters, the parameter can be successfully estimated without exception. The estimation of multiple parameters is less reliable, mainly because the identifiability of the parameters becomes weaker and the problem might have no unique solution. The parameter estimation results are found to be very sensitive to the realization of the initial parameter ensemble, which is mainly related to the use of relatively small ensemble sizes. Increasing ensemble size generally improves the parameter estimation. The quality of parameter estimation also depends on the quality of observation data. It is also found that the results of state estimation are generally improved when simultaneous parameter estimation is performed, even when the estimated parameter values are not very accurate.

Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants

This work studies the effects of sampling variability in Monte Carlo-based methods to estimate very high-dimensional systems. Recent focus in the geosciences has been on representing the atmospheric state using a probability density function, and, for extremely high-dimensional systems, various sample-based Kalman filter techniques have been developed to address the problem of real-time assimilation of system information and observations. As the employed sample sizes are typically several orders of magnitude smaller than the system dimension, such sampling techniques inevitably induce considerable variability into the state estimate, primarily through prior and posterior sample covariance matrices. In this article, we quantify this variability with mean squared error measures for two Monte Carlo-based Kalman filter variants: the ensemble Kalman filter and the ensemble square-root Kalman filter. Expressions of the error measures are derived under weak assumptions and show that sample sizes need to grow proportionally to the square of the system dimension for bounded error growth. To reduce necessary ensemble size requirements and to address rank-deficient sample covariances, covariance-shrinking (tapering) based on the Schur product of the prior sample covariance and a positive definite function is demonstrated to be a simple, computationally feasible, and very effective technique. Rules for obtaining optimal taper functions for both stationary as well as non-stationary covariances are given, and optimal taper lengths are given in terms of the ensemble size and practical range of the forecast covariance. Results are also presented for optimal covariance inflation. The theory is verified and illustrated with extensive simulations.

An OSSE Framework Based on the Ensemble Square Root Kalman Filter for Evaluating the Impact of Data from Radar Networks on Thunderstorm Analysis and Forecasting

Abstract A framework for Observing System Simulation Experiments (OSSEs) based on the ensemble square root Kalman filter (EnSRF) technique for assimilating data from more than one radar network is described. The system is tested by assimilating simulated radial velocity and reflectivity data from a Weather Surveillance Radar-1988 Doppler (WSR-88D) radar and a network of four low-cost radars planned for the Oklahoma test bed by the new National Science Foundation (NSF) Engineering Research Center for Collaborative Adaptive Sensing of the Atmosphere (CASA). Such networks are meant to adaptively probe the lower atmosphere that is often missed by the existing WSR-88D radar network, so as to improve the detection of low-level hazardous weather events and to provide more complete data for the initialization of numerical weather prediction models. Different from earlier OSSE work with ensemble Kalman filters, the radar data are sampled on the radar elevation levels and a more realistic forward operator based on the Gaussian power-gain function is used. A stretched vertical grid with high vertical resolution near the ground allows for a better examination of the impact of low-level data. Furthermore, the impacts of storm propagation and higher-volume scan frequencies up to one volume scan per minute on the quality of analysis are examined, using a domain of a sufficient size. The generally good analysis compared to earlier work indicates that the filter can effectively handle the non-uniform-resolution data on the radar elevation levels. The assimilation of additional data from a well-positioned (relative to the storm) CASA radar improves the analysis of a supercell storm system that uses data from one WSR-88D radar alone; and the improvement is most significant at the low levels. When data from a single CASA radar are assimilated and when the radar does not provide full coverage of the storm system, significant errors develop in the analysis that cannot be effectively corrected. The combination of three CASA radars produces analyses of similar quality as the combination of one WSR-88D radar and one well-positioned CASA radar. The most significant effect of storm propagation speed appears to be on the data coverage, which in turn affects the analysis quality. It is generally true that the more observations, the better the analysis. The results of the EnSRF assimilation are not very sensitive to the propagation speed. The quality of analysis can be improved by employing faster volume scans. The sensitivity of the EnSRF analysis to the volume scan interval is however much less than that of traditional velocity and thermodynamic retrieval schemes, suggesting the superiority of the EnSRF method compared to traditional methods. The very frequent update of the model state by the filter, even at 1-min intervals, does not show any negative effect, indicating that the analyzed fields are well balanced.